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SECTION  1 


INTRODUCTION 

In  October  1991  *  the  Analysis  Section  of  tha  Tactical 
Systems  Development  Branch  (TSDB)  at  MCTSSA  conducted  a  concept 
test  to  determine  if  a  proposed*  work  station  nix  for  an  auto* 
noted  battalion  level  conbat  operations  center  (CQC)  was 
excessive  <see  "Battalion  Conbat  Operations  Center  (COC)  Test 
Report* *  MCTSSA  Oocunent  No.  22T(J<)  1/U-TRP-0 1  dtd  3  Feb  1982). 
Durinp  the  conduct  of  this  test  it  was  noted  by  test  observers 
that  a  significant  number  of  the  test  participants  elected  not 
to  use  the  nap  background  on  the  automated  graphics  display 
terminals  used  for  the  test  < the  participants  had  the  ability 
to  select  or  deselect  the  mas  background  at  will).  A  number  of 
-comments  made  by  the  participants  at  the  conclusion  of  the  test 
indicated  that  the  rationale  for  not  usins  the  mas  background 
fell  into  two  peneral  categories*  Some  of  the  participants 
indicated  that  the  nae  background  simply  was  not  necessary  for 
the  functions  they  were  performing*  Other  participants  indi¬ 
cated  that  the  nap  background  was  not  used  because  its  pres¬ 
ence*  together  with  all  of  the  overlay  symbol opy *  caused  a 
clutter  problem* 

This  unexpected  but  interesting  finding  suppested  that  a 
follow-on  test*  designed  to  investigate  the  functional  utility 
of  the  map  background*  should  be  conducted*  If  it  could  be 
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deternined-  that  the  nap  background  does  not  significantly 
incraasa  tha  aff activanass  of  tha  display  than  it  night  suggast 
that  lass  sophisticated ,  lass  expensive,  lighter  weight  eauip- 
nent  could  be  provided  for  tha  automated  CQC  at  tha  battalion 
level. 

Tha  prinary  ain  of  tha  Craphics  Display  Test,  then,  was  to 
datarnina  whether  or  not  there  is  a  statistically  significant 
diffaranca  batwaan  tha  functional  utility  of  a  graphics  display 
with  a  nap  background  and  a  graphics  display  without  a  nap 
background*  As  a  related  guestion,  tha  Sraphics  Oisplay  Test 
also  investigated  tha  affect  of  display  size  as  it  relates  to 
tha  functional  utility  of  a  graphics  display* 


SECTION  2 


DESCRIPTION 

2*1  REFERENCE*  For  a  nor*  detailed  discussion  of  test  design* 
test  procedures  and  data  analysis  methodology,  see  "Graphics 
Display  Test  Plan"  (MCTSSA  Document  No*  22T901/U-TP-01  dtd 
1  Fed  1982). 

2*2  PURPOSE*  The  purpose  of  the  Graphics  Display  Test  was  to 
assess  the  effects  of  the  presence/absence  of  a  nap  background 

m 

and  of  display  size  as  they  relate  to  the  functional  utility  of 
the  graphics  display  for  the  battalion  level  Tactical  Combat 
Operations  (TCQ)  supported  Combat  Operations  Center  (COC) * 

2*3  PROCEDURES 

Four  display  types  (designated  type  A,  3*  C  and  0)  were 
tested*  Type  A  was  a  large  display  (approximately  10*3/4“  by 
10-3/4" )  with  a  nap  background*  Type  B  was  a  large  display 
without  a  map  background*  Types  C  and  D  were  small  displays 
(approximately  4-1/2"  by  4—1/2")  with  and  without  a  nap  back¬ 
ground  respectively*  Each  display  type  was  utilized  for  twelve 
test  iterations  so  that  each  level  of  the  two  factors  (nap 
background  and  size)  were  tested  twenty-four  times* 

Test  participants  consisted  of  twenty  Narine  Corps  and 
four  U*  3*  Army  officers  holding  primary  ground  combat  MOSs 
and/or  current  or  recent  tactical  experience  by  virture  of 


formal  schooling  or  billet  assignment.  Each  participant  want 
through  two  test  iterations  utilising  two  of  the  four  display 

typos. 

• 

Each  tost  itoration ' was  conducted  against  ono  of  two  sce¬ 
nario  variations  <a  different  scenario  was  required  for  each 
iteration  that  a  particular  participant  performed) .  Both  sce¬ 
narios  involved  a  reinforced  infantary  battalion  conducting 
independent  operations  within  an  assigned  tactical  area  of 
responsibility  (TAGR) . 

2.4  OBJECTIVES 

Two  objectives  were  established  for  the  test:  evaluation 
of  the  effectiveness  of  the  four  display  types  in  providing 
assistance  to  the  commander  and  assessment  of  the  user  compat¬ 
ibility  of  the  various  displays. 

Effectiveness  evaluations  involve  measurements  of  observ¬ 
able  phenomena  and  produce  numerical  results  which  are  directly 
obtainable.  Two  effectiveness  indicators  were  measured  during 
the  Graphics  Display  Test: 

-  Cumulative  time  required  to  obtain  six  pieces  of 
tactical  information 

-  Total  score  received  on  the  answers  to  six  tactical 
questions  as  a  function  of  the  graphics  display 


utilized 


Compatibility  assessments  involvp  elicitation  of  test 
participant  appraisals/opinions  and  do  not  always  yield  quan- 
titative  results*  In  those  cases  where  quantitative  results 
are  obtained*  it  is  usually  through  indirect  means*  Two  com¬ 
patibility  indicators  were  used  during  the  Graphics  Display 
Test: 


-  Participants'  ordinal  judgements  at  the  displays 

-  Subjective  comments/opinions  expressed  by  the 
the  participants 

2*3  GENERALIZED  TEST  FACILITY*  The  capabilities  at  the 
trophies  diseley  equipment  utilized  by  an  automated  battalion 
C8C  were  simulated  by  MCTSSA's  Generalized  Test  Facility  (GTF) » 
Although  the  GTF  graphics  terminals  do  not  physically  resemble 
the  hardware  envisioned  for  MTACCS*  they  can  be  configured/ 
controlled  to  elosely  simulate  key  characteristics  of  this 
hardware*  Far  this  test*  a  hi qh  resolution  diqitalized  map  was 
utilizad  to  simulate  the  presence  of  a  paper  nap  inserted 
behind  a  transparent  qraphics  display*  Additional  graphics 
capabilities  of  the  GTF  equipment  was  utilized  to  produce 
monochromatic  red  overlays  consisting  of  standard  military 
synboloqy  for  units  and  control  measures*  The  symbolo9s  for 
applicable  units  was  made  to  move  across  the  display  in  near 
real-time  {simulation  of  PLP3)  in  accordance  with  the  scenario 


SECTION  3 


RESULTS 


3*1  GENERAL 

This  section  describes  the  results  used  to  satisfy  the 
objectives  stated  in  Section  2*  The  data*,  together  with  sun- 
nary  statistics  and  interpretations  as  to  the  statistical  sig¬ 
nificance  of  the  data*  are  presented* 

In  order  to  facilitate  the  presentation*  the  following 
definitions  and  conventions  are  appropriate*  Analysis  of  var¬ 
iance  (ANGVA)  is  a  connon  statistical  technique  which  tests  the 
hypothesis  that  there  is  no  statistical  difference  between  the 
nean  value  of  data  drawn  fro*  two  or  nore  populations*  The 
ANQVA'  procedure  results  in  a  statistic  called  the  F-ststistic* 
Corresponding  to  the  F-statistic  is  a  P -value  (or  significance 
level)  which  is  an  expression  of  the  probability  that*  if  the 
hypothesis  is  rejected*  you  are  rejecting  the  true  case* 
Rejection  of  a  true  hypothesis  is  called  "type  I  error"*  It  is 
inportent  to  realize  that  the  probability  of  type  I  error  is  a 
conditional  probability*.  i*e»*  it  only  has  neaning  if  the 
hypothesis  is  rejected*  It  is  not  correct  to  say  that*  if  the 
hypothesis  is  not  rejected*  you  have  a  probability  of  one  ninus 
the  P-velue  of  being  correct* 
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For  the  purposes  of  this  discussion#  a  difference  between 
the  measures  of  performance  for  the  four  display  types  will  be 
considered  statistically  significant  only  if  the  probability  of 
type  X  srror  is  equal  to  or  lass  than  one-tenth.  In  othar 
words#  tha  hypothasis  will  ba  rejected  only  if  thara  is  no  nor# 
than  a  tan  parcant  chanes  of  being  wrong  in  doina  sc* 

3.2  EFFECTIVENESS  RESULTS 

Two  naasuras  of  perfornance  CHOP)  wara  utilized  to  eval- 
uata  tha  affactivanass  of  tha  displays  in  providina  asssistanca 
to  tha  eonnandar*  Tha  first  naasura  (MOP  la)  was  tha  total 
aneunt  of  tine  (in  seconds )  required  to  obtain  six  piacas  of 
tactical  inf or nation  fro*  tha  display*  Tha  second  naasura  (MOP 
lb)  was  tha  total  scora  achiavad  on  tha  answars  given  to  six 
tactical  questions  as  a  function  at  tha  display  bain*  utilized* 
Tha  results  of  thasa  naasuras  are  contained  in  Tables  3-1  and 
3-2  respect ivaly. 

For  both  naasuras  of  par for nance*  three  hypotheses  wara 
established*  Tha  three  hypotheses  ware;  that  no  statistically 
significant  difference  existed  between  a  large  and  a  snail 
display  size*  that  no  statistically  significant  difference 
existed  between  a  display  with  and  a  display  without  a  nap 
backgorund#  and  finally#  that  there  was  no  significant  inter¬ 
action  effect  between  display  size  and  tha  presence  or 
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TABLE  3-1 


TOTAL  TIME  REQUIRED  TO 
OBTAIN  TACTICAL  INFORMATION 
<MOP  la) 


Lsr*a  Display 

Lar*e  Display 

with 

-  without 

Map  Backarnd 

Map  3aek*rnd 

342 

338 

314 

371 

133 

487 

23d 

204 

373 

331 

288 

432 

483 

4?4 

388 

.  234 

422 

381 

333 

443 

434 

4®1 

38® 

404 

Avars*#*? 

34? 

428 

Avar**#  with  Map  Background? 
Avers*#  without  Map  Background? 
Avar**#  for  Lar*a  Display? 
Avar**#  for  Snail  Display? 
Overall  Avars*#? 


Snail  Display  Snail  Display 


with 

Map  3sck*rnd 

without 
Map  Backarnd 

310 

377 

293 

478 

233 

409 

444 

344 

191 

247 

279 

274 

310 

440 

222 

332 

344 

374 

708 

410 

237 

240 

308 

370 

343 

433 

347 

431 

38? 

38? 

38?  Std*  Deviation ?  134 


Not#?  Tina#  ar#  to  nearest  whole  second 


? 


TABLE  3-2 

TOTAL  SCORE  ON 
TACTICAL  QUESTIONS 
(MOP  IB) 


Lars*  Display 
witfi 

Map  Backqrnd 


Lars*  Display 
without 
Map  Backsmd 


7 

4- 

6 

4: 

5 

2i 

4 

Z 

Snail  Display 
with 

Map  Baeksrnd 


Snail  Display 
without 
Map-  Backsrnd 


Avar as**: 

37*3  37.2 

41 .6 

38.6 

3*. 3 

Avara**  without  Map  Background 

t  37.? 

Avar as*  for  Larsa  Display: 

37.3 

Avarasa  for  Snail  Display: 

40.1 

Ovarall  Avarasat 

38.7  Std. 

Daviation 

Not#:  Ransa  of  posaifrla  acoraa 

wa*  6  to  60 

or  absence  of  a  nap  background.  Table  3-3  contains  the  ANQVA 
Tabla  for  the  first  Measure  of  performance  and  Table  3-4  is  the 
AWOVA  Table  for  the  second  measure  of  performance. 

Beseb  on  the  convention  stated  in  paragraph  3.1.  only  one 
statistically  significant  difference  between  display  types  was 
indicated.  Specifically,  a  statistically  significant  dif¬ 
ference.  as  measured  by  the  mean  time  required  to  obtain  tac¬ 
tical  information  (MOP  la) .  was  found  to  exist  between  the 
with  and  the  display  without  a  map  background.  The 
presence  or  absence  of  the  map  background  was  not  a  significant 
factor  when  measured  by  total  score  received  (MOP  lb).  Display 
size  was  not  a  significant  factor  when  measured  by  either  of 
the  MOPs  and  in  both  cases  there  was  not  a  significant  inter¬ 
action  effect. 

3.3  COHPATXBXLmr  RCSULTS 

Two  measures  of  user  compatibility  were  utilized.  The 
first  measure  (M0P2a)  was  the  ordinal  judo aments  of  the  par¬ 
ticipants  as  to  their  assessment  of  the  compatibility  of  the 
four  displays.  The  second  measure  (MOP  2b)  was  the  subjective 
comments/ op inions  expressed  by  the  participants. 


The  first  measure  was  quantified  by  constructing  an 
interval  seals  from  the  ordinal  judaenents  (see  "Graphics  Dis¬ 
play  Tast  Plan"  for  details)*  Tha  ordinal  judaenents  ara  sum¬ 
marized  in  Table  3-5  and.rasultad  in  tha  following  interval 

scale: 

d  b  c  A 

- , — i - 1 - 1 - 

Tha  two  obvious  aroupinas  ara  display  types  A  and  C  at  tha 
hi  ah  and  of  tha  scale  and  display  types  9  and  D  at  tha  lower 
and  of  tha  scale*  This  result  indicates  that  tha  participants 
fait  that  tha  displays  with  a  nap  background  ware  no re  compat¬ 
ible  than  tha  displays  without  a  nap  back around  and  that  within 
each  of  these  aroupinas*  tha  larae  display  was  preferable  to 
tha  snail  display*  An  additional  observation  is  that  tha 
interval  between  C  and  9  is  araater  than  either  of  tha  intra— 
aroup  intervals*  This  is  additional  evidence  that  tha  partic¬ 
ipants  considered  the  nap  beckaround  to  be  the  dominant  factor 
in  deterninina  the  user  compatibility  of  the  display* 

Tha  Kendall  coefficient  of  concordance  was  utilized  to 
tast  the  hypothesis  that  the  participants  were  inconsistent  in 
establishina  their  ordinal  rankinas*  Tha  results  were  that 
this  hypothesis  can  be  rejected  with  near  certainty  (P-valua 
less  than  *0001)* 
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TABLE  3-3 


PARTICIPANTS'  ORDINAL  JUDGEMENTS 

This  tabla  raflacts  the  nunber  of  participants  who  ranked 
display  "1"  above  display  "j" 

i  * 

J 

A 

B 

C 

D 

No tar  Whan  crosa  diagonal  elanent*  do  not  sun  to  24*  aitnar 
a  tia  or  a  failure  to  draw  a  conparison  is  indicated 


The  second  neesur*  of  user  compatibility  was  quantified 
only  to  the  extent  that  percentage*  of  participants  expressing 
a  connon  opinion-  were  identified*  The  following  paragraph*  are 
a  sunnary  of  the  participants'  written  connents*  They  are 
presented  in  descend ins  order  of  the  frequency  with  which  the 
connent  was  nade* 

The  nest  freguent  connent  nade  was  an  expression  of  a 
strons  preference  for  a  display  with  a  nap  background* 
Approxinately  sixty  percent  of  the  participants  expressed  this 
opinion*  Of  those  who  save  a  rationale  for  this  opinion*  the 
nost  frequent  was  the  difficulty  of  correlating  infornation 
between  the  display  < without  a-  nap  background)  and  the  paper 
nap* 

Approxinately  twenty— five  percent  of  the  participants 
expressed  the  opinion  that  the  "ideal"  display  would  give  the 
user  the  ability  to  call  up  or  eliminate  the  nap  background  at 
will* 

Approxinately  twenty  percent  of  the  participants  objected 
to  the  nonochr onat i c  red  overlays*  The  nost  fresuent  rational* 
given  was  the  tendency  for  the  red  overlay  infornation  to  blend 
into  the  nap  colors*  One  participant  felt  that  the  red  syn- 
bology  would  present  a  serious  problem  in  a  red  light  environ¬ 
ment  during  night  operations* 


A  strong  preference  for  th®  large  display  siz®  was 


•xpr®ss®d  by  sixt«®n  p«rc®nt  of  th®  participants* 

Approxinat®ly  eight  percent  of  th®  participants  expressed 
a  strong  pr®f®r ®nc®  for  a  display  without  a  nap  background* 

Th®  rational®  was  that  a  nap  background  is  not  usually  neces¬ 
sary  for  fir®  support  coordination  r®lat®d  functions  and  that 
whan  datailad  terrain  study  is  necessary*  it  is  ®asi®r  to  per- 
forn  using  a  papar  nap* 

On*  participant  (approxinately  four  p®re®nt)  axprassad  th® 
opinion  that  a  display  that  could  present  only  a  few  nap  high- 
.lights*,  i»e**  kay  terrain*,  roads*  etc.*  would  nak®  it  unn®c®s- 
sary  to  have  a  detailed  nap  background*  Th®  rationale  was  that 
a  few  nap  highlights  would  nak®  th®  correlation  of  infornation 
between  th®  display  and  an  acconp anying  paper  nap  nuch  easier 
to  acconp lish* 
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SECTION  4 


DISCUSSION 

4*1  EFFECTIVENESS  EVALUATIONS 

The  analysis  for  Objective  1  uncovered  a  statistically 
significant  diffarsncs  in  tbs  amount  of  tins  required  to 
extract  tactical  information  from  a  display  with  a  map  back¬ 
ground  as  compared  to  a  disp^lay  without  a  map  background* 

This  was  trus  irrespective  of  the  size  of  the  display* 

It  is  important  to  understand  that  the  term  M statistically 
significant  difference*  means  that  there  is  strong  statistical 
evidence  that  the  measured  difference  in  the  mean  time  to 
extract  information  is  a  true  difference  as  opposed  to  a  random 
occurence*  That  the  difference  is  statistically  significant 
does  not  necessarily  mean  that  the  difference  is  operationally 
significant*  Additional  analysis  may  aid  the  reader  in  making 
a  determination  as  to  the  operational  significance  of  the 
results* 

The  difference  in  the  measured  mean  time  required  to 
extract  information  from  the  map  background  display  as  compared 
to  the  no  map  background  display  was  eighty— four  seconds  or  a 


twenty  percent  decrease  in  tine*  A  two  sided,  ninety  percent 


confidence  linit  for  til*  difference  between  t he  two  n«an  tm«« 
is: 

Probability  <  1?  s«c  <  True  Difference  <  149  sec  )  *  *9 

This  represents  a  range  of  reduction  in  tin*  of  between 
four  percent  and  thirty— five  percent  whan  conpared  to  tha  warn 
tin*  for  a  no  nap  background  display*  Statad  anothar  way,  tha 
probability  that  tha  prasanca  of  tha  nap  background  rasults  in 
a  tina  raduction  of  batwaan  four  and  thirty— fiva  parcant  is 
nina— tenths#  Tha  naxinun  likelihood  astinata  for  tha  tina 
raduction  is  twenty  parcant* 

Tha  data  analysis  for  Objective  1  found  no  other  statis¬ 
tically  significant  diffarancas  batwaan  display  types*  That  is 
to  say,  tha  statistical  evidence  is  that  all  other  observed 
differences  ware  randan  occurences  and  do  not  reflect  any  true 
difference*  Tha  inplication  is  that,  while  the  test  partici¬ 
pants  ware  able  to  work  faster  with  tha  display  types  that 
included  a  nap  background,  they  ware  not  able  to  work  any  nor a 
accurately  with  one  display  as  conpared-  to  another* 

4.2  COMPATIBILITY  ASSESSMENTS 

A  high  degree  of  consistency  existed  between  tha  two 
neasures  of  par for nance  utilized  for  an  assessnant  of  tha  user 


IS 


compatibility  of  tha  displays*  Both  tha  ordinal  judgamants  and 
tha  writtan  commants  of  tha  participant  indicata  tha  baliaf 
that  tha  prasanca  of  a  map  background  on  tha  graphics  display 
anhancaa  tha  usar  compatibility  at  tha  display* 

4*3  LIMITED  SCOPE  OF  TEST 

This  tast  was  conductad  as  a  follow-on  tast  to  tha 
Battalion  Combat  Oparations  Cantar  <COC)  Tast  and  was  dasignad 
to  addrass  tha  spaeific  issua  statad  in  paragraph  2*2  of  this 
raport*  Within  tha  limitad  scopa  of  this  tast*  tha  rasults 
indicatad  soma  advantaqa*  of  a  graphics  display  that  has  tha 
capability  at  including  a  map  background*  Issuas  such  as  cost* 
systam  davalopmant  tima*  systaa  mobility  at  tha  battalion  laval 
and  many  othars  wars  not  considarad* 


